model(deepseek-v4): DeepSeek-V4-Pro is the SAME architecture as V4-Flash — gate the config descent (#504) - #511
Merged
Merged
Conversation
…ash -- gate the config descent's shape-generality (#504) `deepseek-ai/DeepSeek-V4-Pro` is `DeepseekV4ForCausalLM` / `model_type: deepseek_v4`. A key-by-key diff of both shipped `config.json` files has ZERO new and ZERO removed keys: hidden 4096->7168, layers 43->61, heads 64->128, experts 256->384, moe_inter 2048->3072, q_lora 1024->1536, o_groups 8->16, index_topk 512->1024, routed_scaling 1.5->2.5, and compress_ratios[0:2] [0,0]->[128,128]. Everything else is identical, including head_dim 512, hc_mult 4, num_hash_layers 3, sqrtsoftplus/noaux_tc and expert_dtype fp4. No architecture work is owed: Pro is a CONFIG VARIANT of this row, not a new model row. Grounded three ways rather than asserted. Upstream (pin 555967922, vLLM 0.26.0.dev0): one `vllm/models/deepseek_v4/` package serves both, every dimension read off `config` (`nvidia/model.py:535-679`, `attention.py:193-211`, `:708`). Compressor presence is `compress_ratio > 1` (`attention.py:334`) over `max(1, compress_ratios[layer_id])` (`:209`), and the compressor asserts `compress_ratio in [4, 128]` (`compressor.py:171`). Ratio 0 maps to 1 and so means NO compressor -- the same polarity as our `has_compressor(l) = ratio != 0` (`deepseek_v4.h:127`). Read the whole ~38-file package for literal Flash dimensions: none. Real checkpoints: normalizing layer/expert indices out of both `model.safetensors.index.json` files gives 98 distinct tensor-name patterns on each side with ZERO unique to either (Pro 145116 tensors, Flash 69187). Predicting compressor/indexer layers from `compress_ratios` alone reproduces both exactly -- Flash 41/21, Pro 61/30 -- and Pro's layers 0-1 carry the 4-tensor ratio-128 compressor group where Flash has none. Our side: the only shape assertion in 6930 lines of V4 sources is `head_dim == 512` (`deepseek_v4_weights.cpp:173`), which Pro satisfies; all dims come off config (`deepseek_v4.cpp:1725-1731`) and all CUDA shared memory is `extern __shared__`. The new CPU gate (5 cases / 197 assertions, no fixture, no GPU) drives `ParseDeepseekV4Params` with the real Pro values and the REAL compress_ratios arrays, asserts the 61/30 and 41/21 counts and the layer-0/1 delta, and carries Flash as the control arm so nothing can be satisfied Flash-shaped. RED evidence by mutation, tree restored byte-for-byte (`md5sum -c` OK) after each: hardcoding `num_hidden_layers = 43` reds 3 cases / 5 assertions; adding `layer >= 2 &&` to `has_compressor` reds 2 / 3; hardcoding `o_groups = 8` reds 2 / 3. Restored: 5/5 197 SUCCESS. All nine V4 suites green on the same build -- scaffold 62, dsa 38, compressor 164, moe 716, mhc 125, forward 26, mtp 29, gguf_load 931, pro_variant 197. Hardware verdict recorded, not worked around: 1.599T params is 805 GiB native, 864 GiB for `nvidia/DeepSeek-V4-Pro-NVFP4`, and 534.7 GiB for the smallest real GGUF (`teamblobfish` Q2_K-XL). One GB10 is 119 GiB, so Q2 is ~4.5x over and the ~400 GB 2-bit floor means no quantization closes it. The pinned oracle needs the same memory, so Pro's load/forward/strict gates are PENDING an external resource -- a named blocker, not a correctable finding. No lifecycle transition, so no STATUS/BENCHMARKS write is owed. Docs surfaces are untouched deliberately: no user-visible capability changed, because Pro cannot be run on this hardware. Filed while assessing this: #505 -- `DsaTopkKernel` sizes `chosen[512]`/ `picked[64]` by literal while `index_topk` is 512 (Flash) / 1024 (Pro). Latent today only because `dsa_dense = (be.gguf != nullptr)` (`deepseek_v4.cpp:668`) forces the indexer off on the real path; it becomes a silent thread-stack overflow once the real-geometry DSA residual lands. Spec section 5 records the upstream candidate-window contract (`indexer.py:270-290`, full causal prefix in COMPRESSED-key space) for whoever builds that out. `test_cpu_x86_llamacpp_floor` is red in preflight from box contention, not from this change: the pristine base SHA fdd4526 fails the same test with NO_QUIET_WINDOW at load 43+ driven by two concurrent sessions' builds, and the harness reads none of the four paths here. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
No overlap with the four paths this branch touches; re-gated after the merge. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
No overlap with the four paths this branch touches; re-gated after the merge. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
added a commit
that referenced
this pull request
Aug 12, 2026
…_topk -- remove the literal bounds (#505) Lands the #505 fix: DsaTopkKernel sized `bool chosen[512]` and `int64_t picked[64]` by literal while index_topk is 512 (V4-Flash) / 1024 (V4-Pro). MEASURED on dgx.casa (GB10, sm_121a): the pre-fix kernel at Flash's own width takes `cudaStreamDestroy: an illegal memory access` and SIGABRTs; after the two-pass threshold rewrite the suite is 23/23 with 83913/83913 assertions and 0 skipped, re-verified from the merged tree with CUTLASS + FlashAttention-2 hard-verified in that run's own configure log. The fix removes the bounds rather than asserting them, so there is no configurable limit left to outgrow, and it drops the O(topk^2) emit sort. FRESH REVIEW: PASS. An independent reviewer reproduced both the defect and the fix on real sm_121a hardware, fuzzed 3,000,081 shapes across three independent implementations (host reference, kernel transcription, and its own O(n^2) rank-count oracle) with ZERO divergence, and ran a 12-row device mutation table. It found no blocking issue and 6 non-blocking findings, addressed in a follow-up. Notably it confirmed the tie-heavy case is uniquely load-bearing: it is the only case that catches a tie-break inversion or a value-only threshold. CI: windows-msvc-cpu and windows-msvc-vulkan are red at the repo-wide BASELINE, not from this change -- the same two jobs fail on #539, #541 and on #511 which already merged, and this PR's own log shows every target building with no `error C####`, failing instead in the release-packaging PowerShell step. Merged on direct developer instruction after the requested review pass. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
added a commit
that referenced
this pull request
Aug 13, 2026
…opk, and correct two guard-comment overclaims (#552) Lands the six findings from the fresh review of #505, which returned PASS after failing to find any input where the two-pass threshold selection diverges from `DsaTopkSelect` -- 3,000,081 fuzzed shapes across three independent implementations, zero divergence. CODE: `DsaTopkLaunch` now checks `cudaGetLastError()` after the launch and refuses non-positive `topk` to match the host reference, which previously asserted while the device arm silently returned an empty vector. TESTS: two new device cases. One drives the window clamps -- under-run, over-run, both at once, and an in-range control -- which nothing exercised before. One asserts BOTH arms refuse topk=0 and topk=-1. PROSE: the `w < topk` guard comment claimed to defend against a NaN row writing past its own row. That cannot occur: pass 2's predicate is satisfied by exactly rank(th) <= topk elements and a NaN never satisfies it either. Corrected to belt-and-braces, bounded by construction. "Strictly cheaper" is now scoped to a comparison with the old kernel, not fitness for the real geometry. REFUTED IN FLIGHT: finding 4 held that the launch check would make the next fault attributable to this launch. The arm built to demonstrate it disproved it -- the pre-fix kernel WITH the check still reports `cudaStreamDestroy: an illegal memory access`, because a stack-overflow fault is asynchronous while `cudaGetLastError()` covers launch-configuration errors. The check is kept and its comment now says what it actually does; claiming otherwise would have repeated finding 1's defect inside the change that exists to correct one. DEVICE ARMS on dgx.casa (GB10 sm_121a), each a fresh nvcc rebuild from a pristine kernel with the binary mtime verified to advance, scoped with `-tc=` never `-ts=`, under `flock $HOME/gpu.lock`: baseline 6/6 SUCCESS; no_topk_guard FAILURE; no_ws_clamp exit=134 CRASHED; no_we_clamp 1052 failed assertions; no_launch_check SUCCESS (unobservable by construction); prefix_with_check exit=134; restored full suite 25/25 with 90062 assertions, 0 skipped, SUCCESS. Both new cases have teeth against the mutation each was written for. CPU 25/25 and 13/13, preflight fully clean including the contention-sensitive floor harness. Windows MSVC red is the repo-wide baseline (same two jobs fail on #539, #541 and on #511 which already merged; no `error C####`, the failure is the release-packaging PowerShell step). Merged on direct developer instruction. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #504. Spec:
.agents/specs/deepseek-v4-pro.md. Row:MODEL-TEXT-deepseek-v4-deepseek-v4-for-causal-lm.What this establishes
deepseek-ai/DeepSeek-V4-ProisDeepseekV4ForCausalLM/model_type: deepseek_v4— the same architecture as DeepSeek-V4-Flash, which this row already supports. No architecture work is owed. Pro is a config variant of this row, not a new model row, and this PR adds the gate that keeps that claim honest.A key-by-key diff of both shipped
config.jsonfiles has zero new and zero removed keys. Every difference is a scaled value: hidden 4096→7168, layers 43→61, heads 64→128, experts 256→384, moe_inter 2048→3072, q_lora 1024→1536, o_groups 8→16, index_topk 512→1024, routed_scaling 1.5→2.5,compress_ratios[0:2][0,0]→[128,128].head_dim512,hc_mult4,num_hash_layers3,sqrtsoftplus/noaux_tc,expert_dtype: fp4and the rest are identical. Heads-per-output-LoRA-group is 8 on both.Grounded three ways, not asserted
Upstream (pin
555967922, vLLM 0.26.0.dev0): onevllm/models/deepseek_v4/package serves both models, every dimension read offconfig(nvidia/model.py:535-679,attention.py:193-211,:708). Compressor presence iscompress_ratio > 1(attention.py:334) overmax(1, compress_ratios[layer_id])(:209), and the compressor assertscompress_ratio in [4, 128](compressor.py:171). Ratio 0 maps to 1 and therefore means no compressor — the same polarity as ourhas_compressor(l) = ratio != 0(deepseek_v4.h:127). I read the whole ~38-file package for literal Flash dimensions; there are none.Real checkpoints: normalizing layer and expert indices out of both
model.safetensors.index.jsonfiles gives 98 distinct tensor-name patterns on each side, with zero unique to either (Pro 145116 tensors, Flash 69187). Predicting compressor/indexer layers fromcompress_ratiosalone reproduces both checkpoints exactly:Pro's layers 0-1 carry the 4-tensor ratio-128 compressor group where Flash has none — the single structural difference between the configs.
Our side: the only shape assertion in 6930 lines of V4 sources is
head_dim == 512(deepseek_v4_weights.cpp:173), which Pro satisfies; all dims come off config (deepseek_v4.cpp:1725-1731) and all CUDA shared memory isextern __shared__.The gate
tests/vllm/models/test_deepseek_v4_pro_variant.cpp— 5 cases / 197 assertions, CPU-only, no fixture, no GPU, no download. It drivesParseDeepseekV4Paramswith the real Pro values and the realcompress_ratiosarrays, asserts the 61/30 and 41/21 counts and the layer-0/1 delta directly, and carries Flash as the control arm so no assertion can be satisfied by a Flash-shaped implementation.RED evidence by mutation, tree restored byte-for-byte (
md5sum -cOK) after each:p.num_hidden_layers = 43deepseek_v4_weights.cpp:111has_compressorgainslayer >= 2 &&deepseek_v4.h:127p.o_groups = 8deepseek_v4_weights.cpp:132Restored → 5/5 · 197 SUCCESS. All nine V4 suites green on the same build: scaffold 62, dsa 38, compressor 164, moe 716, mhc 125, forward 26, mtp 29, gguf_load 931, pro_variant 197.
Hardware verdict — recorded, not worked around
1.599T params: 805 GiB native, 864 GiB for
nvidia/DeepSeek-V4-Pro-NVFP4, 534.7 GiB for the smallest real GGUF (teamblobfishQ2_K-XL). One GB10 is 119 GiB, so Q2 is ~4.5x over and the ~400 GB 2-bit floor means no quantization closes it. The pinned oracle needs the same memory, so Pro's load/forward/strict gates are PENDING an external resource — a named blocker, not a correctable finding. Flash already needed 2 Sparks; Pro needs roughly 5-8 at Q2.Gate status
scripts/agent-preflight.sh --stagedat the pushed SHAa532bbe2: every gate ok excepttest_cpu_x86_llamacpp_floor, includingcommit-trailers,trailer suites,doc-checkpoint,now-currentand all record checkers.That one red is box contention, not this change, and I verified it rather than asserting it:
fdd45263— zero changes of mine — fails the same test withNO_QUIET_WINDOW after 30s (busy=118% load=44.97), driven by two concurrent sessions' builds;Docs
No lifecycle transition, so no
STATUS/BENCHMARKSwrite is owed. Docs surfaces are deliberately untouched: no user-visible capability changed, because Pro cannot be run on this hardware.Filed while assessing this
#505 —
DsaTopkKernel(cuda_deepseek_v4.cu:639-641) sizeschosen[512]/picked[64]by literal whileindex_topkis 512 (Flash) / 1024 (Pro). Latent today only becausedsa_dense = (be.gguf != nullptr)(deepseek_v4.cpp:668) forces the indexer off on the real path; it becomes a silent thread-stack overflow once the real-geometry DSA residual lands. Spec §5 records the upstream candidate-window contract (v1/attention/backends/mla/indexer.py:270-290— full causal prefix in compressed-key space) for whoever builds that out.Review note
This branch was authored in a coordinating session, so it has not had an independent review pass. It needs a fresh reviewer on the immutable head before merge — in particular someone to re-run the three mutations above and confirm the restored tree is byte-identical.
🤖 Generated with Claude Code